Add Glossary of Terms to Understanding Airbyte #6235

avaidyanatha · 2021-09-17T23:25:06Z

Main Changes

There's a lot of terms floating around that we can define in one place.

michel-tricot · 2021-09-18T00:20:00Z

docs/understanding-airbyte/glossary.md

+
+**Extract**: Retrieve data from a [source](../integrations/sources), which can be an application, database, anything really.
+
+**Transform**: Clean up the data. This is referred to as [normalization](./basic-normalization.md) in Airbyte and involves [deduplication](./connections/incremental-deduped-history.md), changing data types, formats, and more.


can you put Transform 3rd in the list?

michel-tricot · 2021-09-18T00:20:32Z

docs/understanding-airbyte/glossary.md

+
+### Full Refresh Sync
+
+A **Full Refresh Sync** will attempt to retrieve all data from the destination every time a sync is run. Then there are two choices, **Overwrite** and **Append**. **Overwrite** deletes the data in the destination before running the sync and **Append** doesn't.


Suggested change

A **Full Refresh Sync** will attempt to retrieve all data from the destination every time a sync is run. Then there are two choices, **Overwrite** and **Append**. **Overwrite** deletes the data in the destination before running the sync and **Append** doesn't.

A **Full Refresh Sync** will attempt to retrieve all data from the source every time a sync is run. Then there are two choices, **Overwrite** and **Append**. **Overwrite** deletes the data in the destination before running the sync and **Append** doesn't.

Ah good catch, fixed!

michel-tricot · 2021-09-18T00:21:15Z

docs/understanding-airbyte/glossary.md

+
+### Incremental Sync
+
+An **Incremental Sync** will only retrieve new data everytime the a sync occurs. The first sync will always attempt to retrieve all the data. If the [destination supports it](https://discuss.airbyte.io/t/what-destinations-support-the-incremental-deduped-sync-mode/89), you can have your data deduplicated. Simply put, this just means that if you sync an updated version of a record you've already synced, it will remove the old record.


Suggested change

An **Incremental Sync** will only retrieve new data everytime the a sync occurs. The first sync will always attempt to retrieve all the data. If the [destination supports it](https://discuss.airbyte.io/t/what-destinations-support-the-incremental-deduped-sync-mode/89), you can have your data deduplicated. Simply put, this just means that if you sync an updated version of a record you've already synced, it will remove the old record.

An **Incremental Sync** will only retrieve new data from the source every time the a sync occurs. The first sync will always attempt to retrieve all the data. If the [destination supports it](https://discuss.airbyte.io/t/what-destinations-support-the-incremental-deduped-sync-mode/89), you can have your data deduplicated. Simply put, this just means that if you sync an updated version of a record you've already synced, it will remove the old record.

michel-tricot · 2021-09-18T00:23:10Z

docs/understanding-airbyte/glossary.md

+
+### DAG
+
+DAG stands for **Directed Acyclic Graph**. It's an overly fancy term originally coined by math graph theorists that just describes a tree-like process. For example, in the following diagram, you start at A and can choose B or C, which then proceed to D and E, respectively. This kind of structure is great for representing workflows and is what tools like [Airflow](https://airflow.apache.org/) use to orchestrate the execution of software based on different cases or states.


Suggested change

DAG stands for **Directed Acyclic Graph**. It's an overly fancy term originally coined by math graph theorists that just describes a tree-like process. For example, in the following diagram, you start at A and can choose B or C, which then proceed to D and E, respectively. This kind of structure is great for representing workflows and is what tools like [Airflow](https://airflow.apache.org/) use to orchestrate the execution of software based on different cases or states.

DAG stands for **Directed Acyclic Graph**. It's a term originally coined by math graph theorists that describes a tree-like process. For example, in the following diagram, you start at A and can choose B or C, which then proceed to D and E, respectively. This kind of structure is great for representing workflows and is what tools like [Airflow](https://airflow.apache.org/) use to orchestrate the execution of software based on different cases or states.

you should mention that there can not be a loop inside

We want to make things simple but we shouldn't belittle definitions (overly fancy)!

Updated it - I'll hold back my pot shots at terminology :)

michel-tricot · 2021-09-18T00:28:53Z

docs/understanding-airbyte/glossary.md

+This is only relevant for individuals who want to create a connector.
+{% endhint %}
+
+This refers to how you define the data that you can retrieve from a Source. For example, if you want to retrieve `Account` data from your [Salesforce Source](../integrations/sources/salesforce.md), it needs to be defined clearly so that it can be translated to the destination. Learn more [here](./beginners-guide-to-catalog.md).   


you could to introduce the concept of schema. that would make the defintiion clearer.

Makes sense, I've updated it with a definition I think non-technical people can understand too.

sherifnada · 2021-09-20T03:17:23Z

docs/SUMMARY.md

@@ -193,6 +193,7 @@
  * [Templates](contributing-to-airbyte/templates/README.md)
    * [Connector Doc Template](contributing-to-airbyte/templates/integration-documentation-template.md)
 * [Understanding Airbyte](understanding-airbyte/README.md)
+  * [Glossary of Terms](understanding-airbyte/glossary.md)


this should probably be at the bottom of the list since it's more of a reference rather than something that people read end-to-end

Makes sense, done.

sherifnada · 2021-09-20T03:18:06Z

docs/understanding-airbyte/glossary.md

@@ -0,0 +1,54 @@
+# Glossary of Terms
+
+### ETL/ELT


Can we put this in alphabetical order

Abhi Vaidyanatha added 2 commits September 17, 2021 16:21

Add initial Airbyte Glossary of Terms.

403e5da

Add Glossary to summary.

7824fab

avaidyanatha added the area/documentation Improvements or additions to documentation label Sep 17, 2021

avaidyanatha requested a review from sherifnada September 17, 2021 23:25

avaidyanatha assigned marcosmarxm and unassigned marcosmarxm Sep 17, 2021

avaidyanatha requested a review from marcosmarxm September 17, 2021 23:32

michel-tricot reviewed Sep 18, 2021

View reviewed changes

sherifnada approved these changes Sep 20, 2021

View reviewed changes

Address review comments

48cf5a3

avaidyanatha merged commit 4f0839d into master Sep 21, 2021

avaidyanatha deleted the abhi/glossy-finish branch September 21, 2021 22:50

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Glossary of Terms to Understanding Airbyte #6235

Add Glossary of Terms to Understanding Airbyte #6235

avaidyanatha commented Sep 17, 2021

michel-tricot Sep 18, 2021

avaidyanatha Sep 20, 2021

michel-tricot Sep 18, 2021

avaidyanatha Sep 20, 2021

michel-tricot Sep 18, 2021

avaidyanatha Sep 20, 2021

michel-tricot Sep 18, 2021

michel-tricot Sep 18, 2021

michel-tricot Sep 18, 2021

avaidyanatha Sep 20, 2021

michel-tricot Sep 18, 2021

avaidyanatha Sep 21, 2021

sherifnada Sep 20, 2021

avaidyanatha Sep 20, 2021

sherifnada Sep 20, 2021

avaidyanatha Sep 21, 2021


		Extract: Retrieve data from a [source](../integrations/sources), which can be an application, database, anything really.

		Transform: Clean up the data. This is referred to as [normalization](./basic-normalization.md) in Airbyte and involves [deduplication](./connections/incremental-deduped-history.md), changing data types, formats, and more.


		### Full Refresh Sync

		A Full Refresh Sync will attempt to retrieve all data from the destination every time a sync is run. Then there are two choices, Overwrite and Append. Overwrite deletes the data in the destination before running the sync and Append doesn't.


		### Incremental Sync

		An Incremental Sync will only retrieve new data everytime the a sync occurs. The first sync will always attempt to retrieve all the data. If the [destination supports it](https://discuss.airbyte.io/t/what-destinations-support-the-incremental-deduped-sync-mode/89), you can have your data deduplicated. Simply put, this just means that if you sync an updated version of a record you've already synced, it will remove the old record.


		### DAG

		DAG stands for Directed Acyclic Graph. It's an overly fancy term originally coined by math graph theorists that just describes a tree-like process. For example, in the following diagram, you start at A and can choose B or C, which then proceed to D and E, respectively. This kind of structure is great for representing workflows and is what tools like [Airflow](https://airflow.apache.org/) use to orchestrate the execution of software based on different cases or states.

Add Glossary of Terms to Understanding Airbyte #6235

Add Glossary of Terms to Understanding Airbyte #6235

Conversation

avaidyanatha commented Sep 17, 2021

Main Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment